Website Search Engines  Part 1: How Search Engines Work

Do you know how many Web sites there are on the World Wide Web? Can you even guess? Would you believe that there are currently about 20 million sites on the Internet? * 20,000,000!?! Doesn’t it almost seem miraculous that anyone is able to find what they’re looking for on the Internet? Well, if we didn’t have search engines and directories to help us filter through the massive amount of information out there, it would be a daunting task, indeed. Recent studies have shown that over 85% of Internet users rely on search engines to help them find what they are looking for. So until you have built up awareness of your Web site through traditional media and online marketing, you will depend heavily on search engines to drive traffic to your site.

As a Web site owner and developer, it is vital that you understand how search engines will evaluate the sites that you build for yourself and your customers. This article is the first in a three-part series that will take a look at how search engines work, and what you can do to improve your chances of being ranked well with the major search engines and directories. In the first installment, we are going to give you a brief overview of how search engines build their database of Web sites, and how that information is retrieved in response to user’s queries. Over the next couple of days, you will receive the second and third parts in the series, where we will teach you a few things you can do to optimize your site, and how to promote it with search engines.

How Search Engines Work
Search engines do not actively look through all of the Web sites on the Internet, nor do they maintain an exhaustive list of every single site in existence. Search engines “find” a site when the site owner submits it for inclusion in their database, or through links to it on Web sites that are already in the search engine’s database. Whenever a site is submitted to a search engine, a Web spider or crawler (a software program) indexes all of the information on the site, and checks the links and images so that it can reference your site and its contents. After the search engine updates it’s database (the time frame for this varies—anywhere from every few weeks to every couple of months) and adds your Web site to its listings, your site will be included in the complex relevancy algorithms that are used to determine which sites best match the user’s query. Though the exact set of rules that determine a site’s relevance are a closely guarded trade secret, several of the major search engines use the same database or search engine to power their searches (e.g. search engines that are “powered by Google”).

Ultimately, all search engines share the same goal: maintain a comprehensive catalogue of information to provide useful, relevant search results for their users. The most popular search engines have millions of people using their site everyday, and because of the large audience they reach, they are able to command higher prices for advertising space and premium services. However, if the search engine begins to produce poor, irrelevant results, users will go elsewhere, and their advertising revenue will leave with them. As a Web site owner or developer, it is your job to create and maintain a high quality site that provides vaulable information or services to the search engine’s users.

Now that you understand how search engines populate their databases and respond to user’s queries, you are ready to optimize your Web site and make it more “search engine friendly”. In the next installment, we are going to teach you about description meta tags, title tags, keywords, and making the most our of your text.

*SOURCE: SECURITY SPACE WEB SERVER SURVEY 12/04/2002